The proliferation of automatic faithfulness metrics for summarization has produced a need for benchmarks to evaluate them. While existing benchmarks measure the correlation with human judgements of faithfulness on model-generated summaries, they are insufficient for diagnosing whether metrics are: 1) consistent, i.e., decrease as errors are introduced into a summary, 2) effective on human-written texts, and 3) sensitive to different error types (as summaries can contain multiple errors). To address these needs, we present a benchmark of unfaithful minimal pairs (BUMP), a dataset of 889 human-written, minimally different summary pairs, where a single error (from an ontology of 7 types) is introduced to a summary from the CNN/DailyMail dataset to produce an unfaithful summary. We find BUMP complements existing benchmarks in a number of ways: 1) the summaries in BUMP are harder to discriminate and less probable under SOTA summarization models, 2) BUMP enables measuring the consistency of metrics, and reveals that the most discriminative metrics tend not to be the most consistent, 3) BUMP enables the measurement of metrics' performance on individual error types and highlights areas of weakness for future work.

The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.

Evaluating neural network performance is critical to deep neural network design but a costly procedure. Neural predictors provide an efficient solution by treating architectures as samples and learning to estimate their performance on a given task. However, existing predictors are task-dependent, predominantly estimating neural network performance on image classification benchmarks. They are also search-space dependent; each predictor is designed to make predictions for a specific architecture search space with predefined topologies and set of operations. In this paper, we propose a novel All-in-One Predictor (AIO-P), which aims to pretrain neural predictors on architecture examples from multiple, separate computer vision (CV) task domains and multiple architecture spaces, and then transfer to unseen downstream CV tasks or neural architectures. We describe our proposed techniques for general graph representation, efficient predictor pretraining and knowledge infusion techniques, as well as methods to transfer to downstream tasks/spaces. Extensive experimental results show that AIO-P can achieve Mean Absolute Error (MAE) and Spearman's Rank Correlation (SRCC) below 1% and above 0.5, respectively, on a breadth of target downstream CV tasks with or without fine-tuning, outperforming a number of baselines. Moreover, AIO-P can directly transfer to new architectures not seen during training, accurately rank them and serve as an effective performance estimator when paired with an algorithm designed to preserve performance while reducing FLOPs.

Predicting neural architecture performance is a challenging task and is crucial to neural architecture design and search. Existing approaches either rely on neural performance predictors which are limited to modeling architectures in a predefined design space involving specific sets of operators and connection rules, and cannot generalize to unseen architectures, or resort to zero-cost proxies which are not always accurate. In this paper, we propose GENNAPE, a Generalized Neural Architecture Performance Estimator, which is pretrained on open neural architecture benchmarks, and aims to generalize to completely unseen architectures through combined innovations in network representation, contrastive pretraining, and fuzzy clustering-based predictor ensemble. Specifically, GENNAPE represents a given neural network as a Computation Graph (CG) of atomic operations which can model an arbitrary architecture. It first learns a graph encoder via Contrastive Learning to encourage network separation by topological features, and then trains multiple predictor heads, which are soft-aggregated according to the fuzzy membership of a neural network. Experiments show that GENNAPE pretrained on NAS-Bench-101 can achieve superior transferability to 5 different public neural network benchmarks, including NAS-Bench-201, NAS-Bench-301, MobileNet and ResNet families under no or minimum fine-tuning. We further introduce 3 challenging newly labelled neural network benchmarks: HiAML, Inception and Two-Path, which can concentrate in narrow accuracy ranges. Extensive experiments show that GENNAPE can correctly discern high-performance architectures in these families. Finally, when paired with a search algorithm, GENNAPE can find architectures that improve accuracy while reducing FLOPs on three families.

神经文本排名模型已经见证了显着的进步，并越来越多地在实践中部署。不幸的是，它们还继承了一般神经模型的对抗性脆弱性，这些神经模型已被检测到，但仍未被先前的研究所忽视。此外，Blackhat SEO可能会利用继承的对抗性漏洞来击败受保护的搜索引擎。在这项研究中，我们提出了对黑盒神经通道排名模型的模仿对抗攻击。我们首先表明，可以通过列举关键查询/候选者，然后训练排名模仿模型来透明和模仿目标段落排名模型。利用排名模仿模型，我们可以精心操纵排名结果并将操纵攻击转移到目标排名模型。为此，我们提出了一种由成对目标函数授权的基于创新的基于梯度的攻击方法，以产生对抗性触发器，该触发器会导致有预谋的混乱，而具有很少的令牌。为了配备触发器的伪装，我们将下一个句子预测损失和语言模型流利度限制添加到目标函数中。对通过排名的实验结果证明了对各种SOTA神经排名模型的排名模仿攻击模型和对抗触发器的有效性。此外，各种缓解分析和人类评估表明，在面对潜在的缓解方法时，伪装的有效性。为了激励其他学者进一步研究这一新颖和重要的问题，我们将实验数据和代码公开可用。

在线广告最近已发展成为一个竞争激烈且复杂的数十亿美元行业，广告商在大型和高频上竞标广告插槽。这导致对有效的“自动招标”算法的需求日益增长，这些算法确定了传入查询的投标，以最大程度地提高广告商的目标，但受其指定的约束。这项工作探讨了在日益流行的约束下，为单个价值最大化广告商提供有效的在线算法：返回式增长（ROS）。相对于最佳算法，我们对遗憾进行了量化效率，该算法知道所有查询所有查询都是先验的。我们贡献了一种简单的在线算法，该算法在期望中实现了近乎最佳的遗憾，同时始终尊重指定的ROS约束，当查询的输入顺序为i.i.d.来自某些分布的样本。我们还将结果与Balseiro，Lu和Mirrokni [BLM20]的先前工作相结合，以实现近乎最佳的遗憾，同时尊重ROS和固定的预算限制。我们的算法遵循原始的二重式框架，并使用在线镜像下降（OMD）进行双重更新。但是，我们需要使用非典型的OMD设置，因此需要使用OMD的经典低rebret保证，该保证是用于在线学习中的对抗性环境的，不再存在。尽管如此，在我们的情况下，在更普遍的情况下，在算法设计中应用低纤维动力学的情况下，OMD遇到的梯度可能远非对抗性，但受我们的算法选择的影响。我们利用这一关键见解来显示我们的OMD设置在我们的算法领域中造成了低落的遗憾。

强化学习（RL）为可以在现实世界中自主互动的培训代理提供了潜力。但是，一个关键限制是RL算法对核心超参数和网络体系结构选择的脆弱性。此外，诸如不断发展的训练数据和增加的代理复杂性等非平稳性意味着不同的超参数和体系结构在不同的训练点上可能是最佳的。这激发了Autorl，这是一种试图自动化这些设计选择的方法。一类突出的Autorl方法是基于人群的培训（PBT），这在几个大型设置中导致了令人印象深刻的表现。在本文中，我们介绍了PBT式方法中的两项新创新。首先，我们采用基于信任区域的贝叶斯优化，从而可以全面覆盖高维混合参数搜索空间。其次，我们表明，使用世代相传，我们还可以在一次训练中共同学习体系结构和超参数。利用新的高度可行的Brax物理引擎，我们表明这些创新导致了巨大的性能增长，在即时学习整个配置的同时，大大优于调谐基线。代码可在https://github.com/xingchenwan/bgpbt上找到。

通常针对具有特定模型的特定输入而生成的对抗性示例，对于神经网络而言是无处不在的。在本文中，我们揭示了对抗声音的令人惊讶的属性，即，如果配备了相应的标签，则通过一步梯度方法制作的对抗性噪声是线性分离的。从理论上讲，我们为具有随机初始化条目的两层网络和神经切线内核设置证明了此属性，其中参数远离初始化。证明的想法是显示标签信息可以有效地反向输入，同时保持线性可分离性。我们的理论和实验证据进一步表明，对训练数据的对抗噪声进行训练的线性分类器可以很好地对测试数据的对抗噪声进行分类，这表明对抗性噪声实际上将分布扰动注入了原始数据分布。此外，我们从经验上证明，当上述条件受到损害时，在它们仍然比原始功能更容易分类时，对抗性的噪声可能会变得线性分离。

多视图学习通过LEVERAG-ING-ING-ING相同对象之间的关系来完成分类的任务目标。大多数现有方法通常关注多个视图之间的一致性和互补性。但并非所有这些信息都非常有用于分类任务。相反，它是扮演重要作用的具体辨别信息。钟张等。通过联合非负矩阵分组探讨不同视图中的共同视图中存在的判别和非歧视信息。在本文中，我们通过使用跨熵损耗函数来改善该算法来改善目标函数更好。最后，我们在相同数据集上的原始实施更好的分类效果，并在许多最先进的算法上显示其优越性。

颅内动脉瘤现在是常见的，以及如何智能地检测它们在数字健康方面具有重要意义。虽然大多数现有的深度学习研究专注于医学图像的监督方式，但我们介绍了基于3D点云数据检测颅内动脉瘤的无监督方法。特别是，我们的方法由两个阶段组成：无监督的预训练和下游任务。至于前者，主要思想是将每个点云与其抖动的对应物配对并最大化它们的对应关系。然后，我们设计具有每个分支的编码器和后续公共投影头的双分支对比度网络。至于后者，我们为监督分类和分割培训设计简单网络。公共数据集（内部）的实验表明，我们的无监督方法比某些最先进的监督技术实现了可比或甚至更好的性能，并且在检测动脉瘤血管中最为突出。 ModelNet40上的实验还表明，我们的方法实现了90.79 \％的准确性，这优于现有的最先进的无监督模型。